Changed 'wind_2min_timestamp' variable type from character to date-ti… #52

jeremylweiss · 2023-07-25T22:18:41Z

…me when downloading hourly and daily data.

Changed date-time assignment to 'date_datetime' variable in hourly data. Previously was UTC, now MST

…me when downloading hourly and daily data. Changed date-time assignment to 'date_datetime' variable in hourly data. Previously was UTC, now MST

Aariq

Haven't tested this out, but looks like it's going to do the trick. Remember to add a bullet point to the NEWS.md file about these changes

Aariq · 2023-07-26T16:50:12Z

R/az_hourly.R

+    dplyr::mutate(date_datetime = lubridate::force_tz(lubridate::ymd_hms(.data$date_datetime), tzone = "America/Phoenix")) %>%
+    dplyr::mutate(wind_2min_timestamp = lubridate::with_tz(lubridate::parse_date_time(.data$wind_2min_timestamp, orders = "ymdHMSz"), tzone = "America/Phoenix")) %>%


What's the difference between force_tz() and with_tz()?

My understanding is that both handle date_datetime and wind_2min_timestamp without any warning message while assigning the correct time zone to each variable.

With date_datetime, there is no time zone assigned to values in the database. Using force_tz directly assigns tzone = "America/Phoenix", while keeping dates and hours correct. I tested with_tz here, too, but it assigned UTC as the time zone and the corresponding dates and hours, which is wrong.

With wind_2min_timestamp, there is a time zone already assigned to values in the database. Using with_tz and tzone = "America/Phoenix" correctly assigns the desired time zone while keeping corresponding dates and hours correct.

NEWS.md file updated

Can you commit and push the NEWS.md changes? I don't see them in this pull request

Ah, they're there now. But, I managed to make a conflict

I can show you how to deal with the merge conflict next time we meet (next Friday), or you can pick a time tomorrow or earlier next week: https://calendar.app.google/eCWobQteVLViq5NJ8
It'll be an quick fix, just something easier to show over Zoom than via email.

Aariq · 2023-07-28T19:25:49Z

R/az_daily.R

@@ -86,6 +86,7 @@ az_daily <- function(station_id = NULL, start_date = NULL, end_date = NULL) {
    )) %>%
    dplyr::filter(.data$meta_station_id != "az99") %>%
    dplyr::mutate(datetime = lubridate::ymd(.data$datetime)) %>%
+    dplyr::mutate(wind_2min_timestamp = lubridate::with_tz(lubridate::parse_date_time(.data$wind_2min_timestamp, orders = "ymdHMSz"), tzone = "America/Phoenix")) %>%
    #convert NAs
    dplyr::mutate(
      dplyr::across(


Just below this is code to change values like -9999 to NA, but it currently skips wind_2min_timestamp. I think all you need to do is:

... dplyrr::across( c(wind_2min_timestamp, tidyselect(where(is.numeric))), function(x) ...

Without adding the above code to include wind_2min_timestamp in the conversion of NAs, existing code with wind_2min_timestamp as character automatically converts corresponding nodata values of -99999 to NA while throwing a warning 27 failed to parse, as with

az_daily(start_date = "2023-03-22", end_date = "2023-03-22")

Only one of the 28 stations at that point had wind_2min_timestamp daily values on that date.

I think the nodata value conversion needs to happen somehow in conjunction with character-to-datetime conversion of this variable.

Also, the above code has trouble with wind_2min_timestamp, as it is an object of class "c('POSIXct', 'POSIXt')". This is after the character-to-datetime conversion

@Aariq please review solutions in az_daily() and az_hourly() I just pushed on this branch. I think I figured it out

Aariq

Looks good! Other than an unnecessary (but perfectly functional) across(), this looks great to me.

R/az_daily.R

Aariq · 2023-07-31T14:07:02Z

R/az_daily.R

+    ) %>%
+    dplyr::mutate(
+      wind_2min_timestamp = lubridate::with_tz(
+        lubridate::parse_date_time(.data$wind_2min_timestamp, orders = "ymdHMSz"),


ahhh, I understand this now. So pars_date_time does actually understand the timezone, it just converts it automatically to UTC and with_tz() converts it back to MST. Cool!

Aariq · 2023-07-31T14:09:08Z

R/az_hourly.R

-    dplyr::mutate(date_datetime = lubridate::ymd_hms(.data$date_datetime)) %>%
+    dplyr::mutate(
+      date_datetime =
+        lubridate::force_tz(


but here, ymd_hms() doesn't see a timezone and you need force_tz() to keep the time and only change the timezone. Nice!

Co-authored-by: Eric R. Scott <[email protected]>

Changed 'wind_2min_timestamp' variable type from character to date-ti…

c58cd0d

…me when downloading hourly and daily data. Changed date-time assignment to 'date_datetime' variable in hourly data. Previously was UTC, now MST

jeremylweiss requested a review from Aariq July 25, 2023 22:18

Aariq reviewed Jul 26, 2023

View reviewed changes

Aariq linked an issue Jul 26, 2023 that may be closed by this pull request

Handle new wind variables in API #40

Closed

upgrades to values of date-time variables

50e9c94

Aariq reviewed Jul 28, 2023

View reviewed changes

Merge branch 'main' into 2min-wind-timestamp-variable-type

db866c4

Aariq linked an issue Jul 28, 2023 that may be closed by this pull request

Another possible NA value for temp_soil #55

Closed

Handling of nodata with wind_2min_timestamp

cd866fa

Aariq approved these changes Jul 31, 2023

View reviewed changes

Aariq removed a link to an issue Jul 31, 2023

Another possible NA value for temp_soil #55

Closed

jeremylweiss and others added 3 commits August 4, 2023 11:06

Update R/az_daily.R

8c49f1a

Co-authored-by: Eric R. Scott <[email protected]>

Update R/az_daily.R

db131a1

Co-authored-by: Eric R. Scott <[email protected]>

removing extra across() function

df66f73

jeremylweiss merged commit 6eb7332 into main Aug 4, 2023
6 checks passed

jeremylweiss deleted the 2min-wind-timestamp-variable-type branch August 4, 2023 18:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changed 'wind_2min_timestamp' variable type from character to date-ti… #52

Changed 'wind_2min_timestamp' variable type from character to date-ti… #52

jeremylweiss commented Jul 25, 2023

Aariq left a comment

Aariq Jul 26, 2023

jeremylweiss Jul 26, 2023

jeremylweiss Jul 26, 2023

Aariq Jul 26, 2023

jeremylweiss Jul 26, 2023

Aariq Jul 27, 2023

Aariq Jul 28, 2023

jeremylweiss Jul 28, 2023 •

edited

Loading

jeremylweiss Jul 28, 2023 •

edited

Loading

jeremylweiss Jul 29, 2023

Aariq left a comment

Aariq Jul 31, 2023

Aariq Jul 31, 2023

		dplyr::mutate(date_datetime = lubridate::force_tz(lubridate::ymd_hms(.data$date_datetime), tzone = "America/Phoenix")) %>%
		dplyr::mutate(wind_2min_timestamp = lubridate::with_tz(lubridate::parse_date_time(.data$wind_2min_timestamp, orders = "ymdHMSz"), tzone = "America/Phoenix")) %>%

Changed 'wind_2min_timestamp' variable type from character to date-ti… #52

Changed 'wind_2min_timestamp' variable type from character to date-ti… #52

Conversation

jeremylweiss commented Jul 25, 2023

Aariq left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremylweiss Jul 28, 2023 • edited Loading

Choose a reason for hiding this comment

jeremylweiss Jul 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Aariq left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremylweiss Jul 28, 2023 •

edited

Loading

jeremylweiss Jul 28, 2023 •

edited

Loading